© Journal of Agricultural Education 
Volume 52, Number 4, pp 44-55 
DOI: 10.5032/jae.2011.04044 

Getting to the Bottom Line: How Using Evaluation 
Results to Enhance Extension Programs can Lead to 
Greater Levels of Accountability 

Alexa J. Lamm, Policy Research and Evaluation Specialist 
Glenn D. Israel, Professor 
Amy Harder, Assistant Professor 
University of Florida 

Extension has enhanced the lives ofU.S. citizens through adult education in a myriad of ways. However, 
as budgets get tighter, accountability becomes increasingly more important. Over the years, Extension 
has reported low level impacts rather than the long-term successes that those working within the system 
know are occurring. Without enhanced evaluation-driven environments, Extension systems will continue 
to inadequately report programmatic successes, resulting in a lower perceived public value of Extension 
programs. The use of evaluation can create an atmosphere which encourages organizational thinking 
resulting in the types of accountability reports which are adequate for decision making. This study 
examined how Extension professionals ’ perceptions of evaluation use related to their evaluation 
behaviors. The findings suggest a substantial percentage of Extension professionals are doing just 
enough evaluation to complete mandatory reports. It also showed that Extension professionals who 
valued their own personal use, but not necessarily organizational use of their evaluations, were more 
likely to conduct in-depth evaluations. Recommendations for enhancing evaluation driven work 
environments include clearer communication from administration demonstrating the value of evaluation 
beyond the role it plays in accountability and working directly with Extension professionals to promote 
evaluation-oriented conversations rather than just delivering state-led in-service trainings. 
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Introduction 

Extension is extremely diverse and widely 
distributed, offering the largest adult education 
system in the United States (Franz & Townson, 
2008). It encompasses nearly 3,150 county 
Extension offices, 105 land-grant colleges and 
universities, and the federal government through 
USDA’s National Institute of Food and 
Agriculture (NIFA) (Cooperative State 
Research, Education, and Extension Service, 
n.d.). Extension professionals’ work is 
extremely valuable due to its ability to enhance 
the lives ofU.S. residents by extending the reach 
of the land grant university through research 
based education on topics including agriculture, 
community development, youth development, 
natural resources, nutrition, financial 
management, and horticulture (Rasmussen, 
1989). 


However, due to its size, development, and 
growth over the past hundred years, Extension 
faces some unique challenges when held 
accountable for proving its worth to the public. 
Accountability is a “state of, or process for, 
holding someone to account to someone else for 
something - that is, being required to justify or 
explain what has been done” (Rogers, 2003, p. 
2). Accountability is typically used as a 
justification for conducting evaluations. 
However, accountability driven evaluations 
place an emphasis on looking back and judging 
programs by placing blame or praise and leave 
little room for learning (Patton, 2008). 
Cronbach (2000) believed evaluation is better 
used to gain an understanding of processes and 
outcomes to guide future activities in a positive 
way. Accountability is based on the premise 
that funders will use the information in 
evaluation reports to guide future funding and 
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policy decisions. The information they gather is, 
however, rarely adequate for conducting these 
types of appraisals. In essence “accountability 
systems serve the purpose of providing an 
account of how things are going but not enough 
information to inform decisions or solve 
problems” (Patton, 2008, p. 121). 

Evaluation Use 

Extension’s future is largely dependent upon 
its ability to adapt and learn from its employees 
and environment (Burke, 2008). The use of 
evaluation can create an atmosphere which 
encourages organizational thinking and learning 
resulting in the types of accountability reports 
which are adequate for decision making. 
“Learning means the willingness to go slowly, to 
try things out, and to collect information about 
the effects of actions, including the crucial but 
not always welcome information that the action 
is not working” (Meadows, Randers, & 
Meadows, 2004, p. 7). 

A culture supporting evaluation use within 
most state Extension systems has been limited 
(Radhakrishna & Martin, 1999) despite 
recognition of the importance of evaluation as an 
Extension employee competency (Harder, Place, 
& Scheer, 2010). Due to an initial push to 
measure short-term changes in the 1980s, most 
Extension professionals are assessing their level 
of educational success through results from 
posttests administered at the conclusion of each 
educational activity (Franz & Townson, 2008). 
While immediate reactions and some knowledge 
and skill gains are accounted for through 
posttests, actual behavior changes over time 
along with social, economic and environmental 
impacts of Extension programs are lacking. As 
such, the state and federal Extension systems 
will continue to inadequately report 
programmatic successes over time, resulting in a 
lower perceived public value of Extension 
programs. Identifying and using evaluation 
systems to assess programmatic impacts of 
Extension efforts is part of the National 
Research Agenda: Agricultural Education and 
Communication, 2007-2010 (Osborne, 2007). 
Therefore, a study exploring the ways in which 
Extension professionals’ evaluation behaviors 
are related to personal and organizational 
evaluation use can yield valuable data assisting 
Extension leaders in developing protocols that 


will enhance how evaluations are used system- 
wide. 

Conceptual Framework 

The conceptual framework for this study 
was based on Patton’s (2008) theory of 
utilization focused evaluation. Utilization 
focused evaluators work with intended users “to 
determine priority uses early in the evaluation 
process” (Patton, 2008, p. 98). The chosen 
intended use then informs the design of the 
evaluation process including data collection, 
timing, data analysis, and reporting. Through 
prior planning, the intended use increases the 
chance the evaluation will result in the desired 
impact and be used as intended. Recognizing 
the different contexts of which the term use can 
have within the practice of evaluation is 
important. Program planners often think they 
are conducting utilization focused evaluation, 
when in fact they are organizing evaluations for 
other purposes under the guise of “use.” Patton 
(2008) recognized there are many distinctions in 
how evaluators, program coordinators, and 
administrators refer to use. These distinctions 
included (a) direct intended uses, (b) longer term 
uses, (c) primarily political uses, (d) misuses, (e) 
non-uses and (f) unintended effects (Patton, 
2008). This study will focus on the direct 
intended uses of evaluation in the context of 
Extension programming. 

Direct intended uses include instrumental 
use, conceptual use, and process use. 
Instrumental use of evaluations includes using 
findings to directly inform decisions and often 
contribute to problem solving (Patton, 2008). 
Instrumental use links the results to an action 
essentially becoming an instrument of action. 
An example of instrumental evaluation use 
would be a basic assessment of an Extension 
program or tool. In Arkansas, a farm pond 
management website was evaluated over a four 
year period (Neal, 2010). Responses to an 
online survey were collected, identifying that the 
web site was stakeholders’ preferred method of 
communication. As a result, the other more 
traditional Extension media used for 
communication with these stakeholders was 
discontinued (Neal, 2010). Another example of 
instrumental use is an evaluation conducted by 
Menalled, Grimberg, and Jones (2009) assessing 
agricultural agents’ needs, knowledge, and 
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interests as they related to sustainable farming. 
The information collected was directly applied 
to the creation of a distance education program 
in sustainable agriculture (Menalled et al., 
2009). 

Conceptual use informs thinking about a 
program or policy. The results of conceptual use 
evaluation are used to assist key people in 
understanding a concept but no action or 
decision can be directly attributed to the results 
(Patton, 2008). While this type of evaluation 
can contribute to offering future conceptual 
insight it is not specific to a certain time or place 
like instrumental use evaluation. Kaplan, Liu, 
and Radhakrishna (2003) conducted a needs 
assessment study in order to plan and develop a 
statewide intergenerational Extension program. 
Kaplan et al. (2003) collected data noting 
Extension professionals’ preferences regarding 
program content and delivery format. The 
results were used to make key decisions about 
the direction curriculum would take and future 
program delivery strategies but were not specific 
to one place in time. Culp and Kohlhagen 
(2004) conducted an evaluative study examining 
Extension professionals’ level of competence 
when working with volunteers. They then used 
this information to plan and develop volunteer 
administration professional development 
opportunities that would increase Extension 
professionals’ knowledge and/or performance in 
the future. 

According to Patton (2008), “Process use 
refers to cognitive, behavioral, program, and 
organizational changes resulting ... from 
engagement in evaluation process and learning 
to think evaluatively” (p. 108). Process use is 
essentially learning from the evaluation process 
itself and engaging in decision-making based on 
the process and not the end result. Rather than 
being results oriented, the evaluation process can 
begin to become a tool for making programmatic 
decisions, as it helps program planners further 
refine and define exactly what they want as an 
intended outcome. Cummings and Boleman 
(2006) worked to identify issues the Texas 
Extension system needed to be addressing 
during the Texas Community Futures Forum by 
engaging stakeholders to gain their input. 
Through this process not only did they identify 
issues of need, but they were also able to engage 
all levels of Extension faculty in conversations 
to deal with common issues, develop issue 


responses, and identify resources needed to 
address these needs (Cummings & Boleman, 
2006). Therefore, not only did the evaluation 
results assist in identifying needs, but the 
evaluation process facilitated open 
communication that would otherwise not have 
occurred. 

An examination of the literature has 
revealed that a focus on evaluation use has the 
ability to enhance programmatic evaluation 
behaviors thereby resulting in more appropriate 
measures for accountability purposes. Through 
a review of the theory of user focused 
evaluation, it has been established that 
individual and organizational use can be 
categorized into three areas: instrumental, 
conceptual, and process evaluation use (Patton, 
2008). By understanding how use is perceived 
and related to evaluation behavior, research 
based recommendations on how to enhance 
evaluation behaviors in the future can be made 
(Patton, 2008). 

Purpose & Research Questions 

The purpose of this study was to examine 
how Extension professionals’ perception of 
evaluation use relates to their evaluation 
behaviors. The research questions guiding this 
study were: 

1. How are Extension professionals 

evaluating their programs? 

2. What are Extension professionals’ 

perceptions regarding the use of 
evaluation? 

3. Do Extension professionals’ perceptions 

regarding the use of evaluation have a 

relationship with the level at which they 
conduct evaluation? 

Methods 

The study presented here is part of a larger 
study and is descriptive and correlational. A 
census of the 128 Extension professionals and 
county directors working directly with 
customers in their respective counties, all within 
the same state, was conducted. The study was 
limited to this state because the Extension 
administration is currently stressing the 
importance of evaluation for accountability 
within the system due to budgetary issues. 
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The target population’s access to the Internet 
and use of e-mail as a communication tool 
enabled the use of an online survey instrument 
(Dillman, Smyth, & Christian, 2008). Due to a 
lack of previous research on the topic, an 
instrument measuring the variables of interest 
was not available, therefore the researcher 
created an organizational evaluation instrument. 
Four sections of the survey instrument were 
germane to the findings in this article: personal 
evaluation behaviors, perceptions of 
organizational evaluation use, personal 
perceptions regarding evaluation use, and 
demographics. 

The instrument was reviewed by an expert 
panel with expertise in Extension and the 
practice of evaluation for content, face validity, 
and survey design. The panel of experts 
represented four universities including North 
Carolina State University, Oklahoma State 
University, Purdue University, and the 
University of Florida. Reliability was calculated 
ex post facto on the two constructs specific to 
this study. The personal perception regarding 
evaluation use construct had a Cronbach’s alpha 
= .82. The perceptions of organizational 
evaluation use construct had a Cronbach’s alpha 
= .83. 

To measure their evaluation behaviors, 
participants were asked to report on how they 
evaluated their best or most important program. 
Participants were asked to respond by marking 
whether or not they had engaged in 27 specific 
data collection or data analysis/reporting 
methods during the past year. If they marked 
they had engaged in the specific method they 
were given a point. The points assigned for each 
item the participants reported engaging in were 
then summed to create an overall behavior score 
which could range from 0 to 27. 

Participants were then asked to assess their 
personal perceptions regarding evaluation use by 
rating a set of six questions related to their 
personal use of the evaluation process on a 
Likert-type scale. The scale ranged from 1 - 
Not at all true for me, 2 - Slightly true for me, 3 
- Somewhat true for me, 4 - Mostly true for me. 


5 - Completely true for me. Three of these six 
items referred to the instrumental use of 
evaluation while the other three referred to the 
conceptual use of evaluation. 

Next, participants were asked to rate seven 
statements related to their perceptions of 
organizational evaluation use on a Likert-type 
scale. The scale ranged from 1 - Strongly 
Disagree, 2 - Disagree, 3 - Neutral, 4 - Agree, 
5 - Strongly agree. Six of these seven items 
referred to instrumental use of evaluation, while 
one referred to the process use of evaluation. 
Finally, participants were asked to identify their 
gender, race/ethnicity, educational level, and 
primary program area. 

Participants were contacted via e-mail using 
Dillman et al.’s (2008) Tailored Design Method 
All of the original 128 e-mail addresses were 
valid. A response rate of 84% (N = 105) was 
obtained. There were 11 responses removed due 
to missing or incomplete data so the useable 
response rate was 73.4%. There were no 
significant differences between respondents and 
non-respondents demographic characteristics; 
therefore the results of the study can be 
generalized to the target population. 

The general demographics collected in the 
survey are displayed in Table 1. Descriptive 
analysis of the demographic data showed that 
there were 52 female (55.3%) and 42 male 
(44.7%) respondents. The large majority 
(86.7%, n = 85) of respondents were 

Caucasian/White (Non-Hispanic) with 

Hispanics representing 9.2% (n = 9). African 
American, Native American, and Other 
categories were represented minimally. The 
majority of respondents (77.8%, n = 70) had 
obtained a Master’s degree while 15.6 {n = 14) 
had a Bachelor’s degree. 

All program areas were represented with 
37.8% (n = 34) of respondents focusing on 4-H 
Youth Development, 23.3% (n = 21) on Family 
and Consumer Sciences, 17.8% (n = 16) on 
Agriculture, and 10.0% (n = 9) on Horticulture. 
The remaining 11.1% (n = 10) of participants 
represented Community Development and 
Natural Resources specializations. 
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Table 1 


Demographics of Respondents 


Characteristic 

77 

% 

Gender 

Female 

52 

55.3 

Male 

42 

44.7 

Race/Ethnicity 

African American 

1 

1.0 

Asian 

0 

.0 

Caucasian/White (Non-Hispanic) 

85 

86.7 

Hispanic 

9 

9.2 

Other 

3 

3.1 

Education Level 

Associate’s Degree 

2 

2.2 

Bachelor’s Degree 

14 

15.6 

Master’s Degree 

70 

77.8 

Ph.D. 

4 

4.4 

Program Area 

4-H Youth Development 

34 

37.8 

Agriculture 

16 

17.8 

Community Development 

6 

6.7 

Family and Consumer Sciences 

21 

23.3 

Horticulture 

9 

10.0 

Natural Resources 

4 

4.4 


Data Analysis 

The first two objectives were addressed 
using descriptive statistics. Responses were 
coded for computer analysis using SPSS. 
Relationships between Extension professionals’ 
evaluation behaviors and their perceptions 
regarding the use of evaluation were described 
by calculating Pearson’s product-moment 
correlation coefficient using Davis’ (1971) 
convention. Magnitude of the relationship is 
noted by Davis (1971) as .01 > r > .09 = 
Negligible, .10 > r > .29 = Low, .30 > r > .49 = 
Moderate, .50 > r > .69 = Substantial, r > .70 = 
Very Strong. 

Results 

Evaluation Behaviors 

Participants were asked to report on how 
they evaluated their best or most important 
program by marking whether or not they had 
engaged in 27 specific data collection or data 
analysis methods during the past year (see Table 
2). The responses were summed to create an 
evaluation behavior score. Overall evaluation 


behavior scores ranged from zero to 27 (M = 
10.53, SD = 6.39). 

When data collection methods were 
reviewed the majority of participants kept 
program participation records (n = 70, 74.5%) 
and used posttests to evaluate specific activities 
(n = 64, 68.1%). Approximately half of the 
participants used posttests to evaluate their 
entire program (n = 51, 54.3%), tracked their 
participants’ gender (n = 50, 53.2%), and used 
interviews to evaluate specific activities (n = 47, 
50.0%). Very few used a comparison group as a 
control when evaluating (n = 5, 5.3%) or used 
tests to evaluate social, economic, or 
environmental (SEE) condition changes (n =17, 
18.1%). Examining data analysis/reporting 
methods revealed the majority of participants are 
reporting the number of customers attending a 
program (n = 72, 76.6%), creating summaries of 
written accounts (n = 55, 58.5%), and reporting 
means and percentages (n = 50, 53.2%). Very 
few used any type of inferential statistic (n = 2, 
2.1%), compared groups (n = 12, 12.8%), or 
reported standard deviations (n = 13, 13.8%). 
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Table 2 


Participants Evaluation Behaviors 


Behavior Items 

77 

% 

Data Collection Methods 

Keep program participation records 

70 

74.5 

Posttest to evaluate activities 

64 

68.1 

Posttest to evaluate entire program 

51 

54.3 

Track participants’ gender 

50 

53.2 

Interviews to evaluate activities 

47 

50.0 

Participant written accounts 

43 

45.7 

Interviews to evaluate entire program 

43 

45.7 

Pre/posttest to evaluate activities 

42 

44.7 

Collect artifacts 

39 

41.5 

Interview to evaluate behavior change 

39 

41.5 

Track participants’ race/ethnicity 

38 

40.4 

Pre/posttest to evaluate entire program 

37 

39.4 

Test to evaluate behavior change 

29 

30.9 

Interviews to evaluate SEE changes 

22 

23.4 

Test to evaluate SEE changes 

17 

18.1 

Comparison group used as a control 

5 

5.3 

Data Analysis/Reporting Methods 

Report actual numbers 

72 

76.6 

Summary of written accounts 

55 

58.5 

Report means or percentages 

50 

53.2 

Summary of artifacts collected 

44 

46.8 

Summary of interview results 

43 

45.7 

Examine change over time 

25 

26.6 

Comparing content of interviews for similarities and differences 

19 

20.2 

Member checking interview results 

19 

20.2 

Report standard deviations 

13 

13.8 

Compare groups 

12 

12.8 

Advanced inferential statistics 

2 

2.1 


Perceptions of Evaluation Use 

Table 3 displays participants’ personal 
perceptions regarding the use of evaluation. 
Using a five-point scale (1 = Not at all true for 
me, 5 = Completely true for me), participants felt 
evaluation was a critical tool for improving 
Extension programs (M = 4.30, SD = 0.84), that 
they identify the needs and interests of their 
stakeholders prior to developing programs (M = 
4.01, SD = 0.85), that they think it is important 
their evaluation results can be used by others in 
their state system ( M = 4.01, SD = 0.82), and 


that they use their evaluation results to make 
decisions about their programs ( M = 4.00, SD = 
0.90). They were not as sure their evaluations 
served the information needs of their community 
stakeholders (M= 3.45, SD = .91) and that they 
report their evaluation procedures and results to 
their community stakeholders ( M = 3.61, SD = 
0.94). Responses to all six perceptions of 
personal evaluation use items were summed and 
averaged to create an overall perception of 
personal evaluation use score (M = 3.90, SD = 
.63) that tended to be positive. 
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Table 3 


Participant’s Personal Perceptions Regarding Evaluation Use 


Evaluation Use Items 

Evaluation Use 
Category 

n 

M 

SD 

I feel evaluation is a critical tool for 
improving Extension programs 

Instrumental 

94 

4.30 

.84 

I identify the needs and interests of my 
stakeholders prior to developing 
programs 

Conceptual 

94 

4.01 

.85 

I think it is important my evaluation results 
can be used by others within my state 
Extension system 

Instrumental 

94 

4.01 

.82 

I use evaluation results to make decisions 
about my programs 

Instrumental 

94 

4.00 

.90 

I report evaluation procedures and result to 
my community stakeholders 

Conceptual 

94 

3.61 

.94 

My evaluations serve the information needs 
of my community stakeholders 

Conceptual 

94 

3.45 

.91 


Note : Scale: 1 = Not at all true for me, 2 = Slightly true for me, 3 = Somewhat true for me, 4 = Mostly true 
for me, 5 = Completely true for me 


Table 4 displays participants’ perceptions of 
organizational evaluation use using a five-point 
scale (1 - Strongly Disagree, 5 - Strongly 
Agree). They were in slight agreement that their 
county directors were interested in using their 
evaluation results ( M = 3.59, SD = 1.04) and 
also slightly agreed that there is a moderate 
interest in using data to make decisions in their 
Extension offices ( M = 3.52, SD = .92). 
Participants did not report a high level of 


agreement when asked to rate the level at which 
their county director ( M = 3.05, SD = 1.12) or 
regional director ( M = 3.07, SD = 1.02) 
communicates about how evaluation results will 
be used. Responses to all seven perceptions of 
organizational evaluation use items were 
summed and averaged to create an overall 
perception of organizational evaluation use score 
(M = 3.22, SD = .74) which was slightly higher 
than neutral. 
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Table 4 


Participant’s Perceptions of Organizational Evaluation Use 


Evaluation Use Items 

Evaluation 

Use Category> 

77 

M 

SD 

My county director is interested in using my 
evaluation results* 

Instrumental 

44 

3.59 

1.04 

There is a strong interest in using data to 
make decisions in my Extension office 

Instrumental 

90 

3.52 

.92 

My regional director is interested in using my 
evaluation results* 

Instrumental 

81 

3.49 

.98 

Extension professionals discuss evaluation 
approaches, challenges, and use in my 
Extension office 

Process 

90 

3.22 

.97 

The state Extension director seeks evaluation 
information when making decisions 

Instrumental 

89 

3.22 

.84 

My regional director clearly communicates 
how evaluation results will be used* 

Instrumental 

81 

3.07 

1.02 

My county director clearly communicates 
how evaluation results will be used* 

Instrumental 

44 

3.05 

1.12 


Note: Scale: 1 = Strongly Disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly Agree. 
*Participants not reporting to a county or regional director did not respond to these items. 


Relationships between Evaluation Behaviors 
and Perceptions of Evaluation Use 

The personal perceptions index was 
moderately correlated with evaluation behavior, 
while the organizational perceptions index had 
only a low correlation with evaluation behavior 
(Table 5). All but one of the personal 
perceptions of evaluation use items were 
moderately correlated with the participants’ 
evaluation behaviors. Whether or not the 
participants’ evaluations served the information 
needs of their community stakeholders (R = .43) 


had the highest correlation among these items. 
Of the organizational perception of evaluation 
use items only one, if Extension professionals 
discuss evaluation approaches, challenges, and 
use in their Extension office (R = .36), had a 
moderate correlation to their evaluation 
behavior. Extension professionals’ perception 
regarding whether or not the state Extension 
director seeks evaluation information when 
making decisions had a negligible negative 
correlation to evaluation behavior (R = -.04). 
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Table 5 

Correlations between Evaluation Behaviors and Perceptions of Evaluation 


Use 


Perceptions of Evaluation Use 


Evaluation behavior 
R _ Magnitude 


Personal Perceptions 

My evaluations serve the information needs of my community 
stakeholders 

I report evaluation procedures and result to my community 
stakeholders 

I use evaluation results to make decisions about my programs 

1 identify the needs and interests of my stakeholders prior to 
developing programs 

I think it is important my evaluation results can be used by others 
within my state Extension system 

1 feel evaluation is a critical tool for improving Extension 
programs 

Organizational Perceptions 

Extension professionals discuss evaluation approaches, 
challenges, and use in my Extension office 

My county director is interested in using my evaluation results 

My county director clearly communicates how evaluation results 
will be used 

My regional director clearly communicates how evaluation 
results will be used 

There is a strong interest in using data to make decisions in my 
Extension office 

My regional director is interested in using my evaluation results 

The state Extension director seeks evaluation information when 
_ making decisions _ 


.47 

Moderate 

.43 

Moderate 

.36 

Moderate 

.35 

Moderate 

.32 

Moderate 

.30 

Moderate 

.28 

Low 

.27 

Low 

.36 

Moderate 

.31 

Moderate 

.29 

Low 

.17 

Low 

.16 

Low 

.14 

Low 

-.04 

Negligible 


Note. Magnitude: .01 > R > .09 = Negligible, . 10 > R > .29 = Low, 30>R> .49 = Moderate, 
.50 >R> .69 = Substantial, R > .70 = Very Strong. 


Conclusions 

This statewide study sheds light on the 
evaluation behaviors that Extension 
professionals are engaging in. It also gives 
insight into how Extension professionals 
perceive their personal use of evaluation 
procedures and results as well as their 
perceptions of how their Extension system as a 
whole values the use of evaluation. This study 
has shown Extension professionals are engaged 
in a wide variety of evaluation behaviors. The 
majority are keeping program participation 
records (74.5%), conducting posttests of their 
activities (68.1%), and conducting posttests on 
their overall programs (54.3%). They are not 
using comparison groups as a control (5.1%) or 
conducting any type of inferential data analysis 
on their results (2.3%). This study supports 


Franz and Townson’s (2008) claim that the 
majority of Extension professionals currently 
utilize posttests given at the conclusion of their 
educational activities to assess the level of 
success. Even though Harder et al. (2010) found 
evaluation as a key competency for Extension 
professionals, this research suggests Extension 
professionals are only engaging in evaluation at 
the most basic level. 

In general, participants tended to believe 
they used evaluation (M = 3.90, SD = .63) but 
had a more neutral perception of organizational 
use (M = 3.23, SD = .74). On a personal level, 
Extension professionals agreed they used 
evaluation instrumentally more than 
conceptually. Therefore, Extension 

professionals are using their evaluation results to 
directly inform their decisions regarding future 
programming rather than to gain future 
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conceptual insight (Patton, 2008). In addition, 
all but one personal perception of evaluation use 
items had a moderate correlation to evaluation 
behavior. The finding that evaluation behavior 
and personal evaluation use are related is 
supportive of Patton’s (2008) previous research 
that a focus on evaluation use has the ability to 
enhance programmatic evaluation behaviors. 
Only two organizational evaluation use items 
had a moderate correlation with the evaluation 
behaviors of Extension professionals. This is in 
direct opposition with previous research 
showing organizational commitment to a 
behavior will create an atmosphere which 
encourages organizational thinking and therefore 
an employee commitment to the behavior 
(Burke, 2008). 

Implications and Recommendations 

This study was designed to examine the 
direct use (Patton, 2008) of evaluation by 
Extension professionals. Recall that direct use 
includes instrumental, conceptual, and process 
use (Patton, 2008). The findings of this study 
provide evidence that suggests Extension 
professionals may use evaluation for 
instrumental use, as indicated by their tendency 
to believe evaluation is a critical tool for 
improving programs. To a lesser extent, 
conceptual use was evidenced as Extension 
professionals mostly thought it was true that 
they used evaluation to make decisions about 
their programs. Both findings are encouraging 
as they indicate a professional commitment to 
delivering quality programs. What may be more 
interesting, and concerning, to note from the 
findings of this study is not the direct use of 
evaluation by Extension professionals but rather 
the implication that Extension professionals in 
this state believe their Extension system as a 
whole uses evaluation for political purposes. 

According to Patton (2008), political use 
includes imposed use and mechanical use. 
Imposed use refers to evaluations conducted in 
response to mandates issued by those in power, 
such as a federal requirement. In fact, there are 
both state and federal requirements for 
Extension to report the number of people 
reached through Extension programming and the 
demographic characteristics of those people. 
Mechanical use occurs when people go “through 
the motions” of evaluation simply to be in 


compliance with evaluation requirements 
(Patton, 2008, p. 112). The high percentages of 
Extension professionals who reported keeping 
program participation records and tracking 
gender, race, and ethnicity as opposed to the 
percentages of Extension professionals who used 
more rigorous evaluation methods such as 
pre/posttests, interviews to evaluate long-term 
outcomes, and control groups are clear 
indicators of imposed and mechanical use. 

While it must be acknowledged that some 
Extension professionals simply lack the 
evaluation expertise to perform behaviors such 
as measuring long-term outcomes or using 
advanced inferential statistics, the findings from 
this study suggest a substantial percentage of 
Extension professionals are doing just enough to 
get by when it comes to evaluation. Franz and 
Townson (2008) also noted a tendency for 
Extension professionals nationwide to use a low 
level of rigor when conducting evaluations. 
Program participation records, post-test only 
designs, and interviews (the quality of which is 
undetermined) are behaviors that require a 
minimal amount of effort, and produce little 
information that can be used to improve 
programs. Yet these are often good enough to 
satisfy organizational accountability 
requirements. The popularity of these methods 
is contradictory to the Extension professionals’ 
purportedly strong beliefs that evaluation is a 
critical tool for improving Extension programs. 
Note the item “critical tool for improving 
Extension programs” R = .28 to evaluation 
behaviors, which is empirical support for a weak 
relationship. This should be cause for concern. 

At the organizational level, Extension 
professionals perceived their county and 
regional directors’ desire to have evaluation 
results to use was more evident than how the 
results would be actually used. Similarly, the 
Extension professionals did not strongly agree 
that their state director used their evaluation 
results when making decisions. Clearer 
communication from administration would be 
helpful in demonstrating the value of evaluation 
beyond the role it plays in accountability. 

The Extension professionals in this study 
made their decisions about using evaluation 
largely independent of what they believed about 
how their county, regional, and state directors 
value and use evaluation. More plainly, 
Extension professionals are likely to do what 
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they are going to do, regardless of what 
administration tells them. Extension instead 
needs to focus on improving its culture of 
evaluation by building upon the positive 
relationship that was observed between 
evaluation use and evaluation discussion. 
Extension professionals who work in offices that 
talk about evaluation are more likely to conduct 
evaluation. Working directly with Extension 
professionals to promote evaluation-oriented 
conversations, rather than just delivering more 
state-led in-service trainings and other top- 
down efforts, is a key step towards building an 
organizational culture that values evaluation. 

Although many barriers exist in engaging 
Extension professionals in the practice of 
evaluation, they can be overcome. In order to 
gain more insight into the issues surrounding the 
challenges of getting Extension professionals to 
increase the amount of evaluations conducted, 
and the level of rigor in which they approach 
evaluating programs, more research should be 


conducted. This study was specific to Extension 
professionals working in a single state and 
generalization beyond the population should be 
done with great caution. A study conducted to 
determine if Extension professionals working in 
other state Extension systems (size, regional 
location, etc.) will assist in a broadening the 
interpretation of the results. 

In addition, a study examining the impact of 
state Extension system structure (i.e. 
professional development offered, 

communication, system-wide culture, work unit 
climate) has on evaluation use will offer 
administrators an idea of what is already 
working within their system and how to enhance 
what is not. In addition, research exploring why 
specific Extension professionals have chosen to 
evaluate their programs with a high level of 
rigor, even though the majority does not, would 
assist in gaining an understanding of how 
banders against evaluation use have been 
overcome. 
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